Structured display shutdown for video pass-through electronic devices

ABSTRACT

The subject technology provides structured-shutdown-mode versions of a video view of a physical setting of an electronic device. An electronic device may provide, at a first power consumption rate, a video view of a portion of the physical setting, the portion being physically blocked from being directly viewed by a user of the device, by the electronic device itself. Structured-shutdown-mode versions of a video view may be provided, at a second power consumption rate lower than the first power consumption rate, to a display of an electronic device when the device determines that it is in a low power or failure state. In this way, the electronic device can reduce the functionality and power consumption of the display in a structured manner, to continue to provide a version of the video view to the user in the low power or failure mode state.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority to U.S. ProvisionalPatent Application No. 63/083,806, filed on Sep. 25, 2020, thedisclosure of which is hereby incorporated herein in its entirety.

TECHNICAL FIELD

The present description relates generally to extended reality settings.

BACKGROUND

Augmented reality technology aims to bridge a gap between virtualenvironments and a physical environment by providing an enhancedphysical environment that is augmented with computer-generated contentthat is not part of the physical environment. As a result, thecomputer-generated content that is not part of the physical environmentappears to be part of the physical environment as perceived by a user.Augmented reality is sometimes provided by a device that provides apass-through video view of the physical environment to a user, and addsthe computer-generated content to the pass-through video view.

BRIEF DESCRIPTION OF THE DRAWINGS

Certain features of the subject technology are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of thesubject technology are set forth in the following figures.

FIGS. 1A-1B depict exemplary systems for use in various extended realitytechnologies, including virtual reality and mixed reality in accordancewith one or more implementations.

FIG. 2 illustrates an example of a physical setting of an electronicdevice that is displaying a video view of a portion of the physicalsetting in accordance with one or more implementations.

FIG. 3 illustrates an example of a structured-shutdown-mode version ofthe video view of FIG. 2 in accordance with one or more implementations.

FIG. 4 illustrates another example of a structured-shutdown-mode versionof the video view of FIG. 2 in accordance with one or moreimplementations.

FIG. 5 illustrates a flow chart of an example process for providingstructure display shutdown for video pass-through electronic devices inaccordance with implementations of the subject technology.

DETAILED DESCRIPTION

The detailed description set forth below is intended as a description ofvarious configurations of the subject technology and is not intended torepresent the only configurations in which the subject technology can bepracticed. The appended drawings are incorporated herein and constitutea part of the detailed description. The detailed description includesspecific details for the purpose of providing a thorough understandingof the subject technology. However, the subject technology is notlimited to the specific details set forth herein and can be practicedusing one or more other implementations. In one or more implementations,structures and components are shown in block diagram form in order toavoid obscuring the concepts of the subject technology.

A person can interact with and/or sense a physical environment orphysical world without the aid of an electronic device. A physicalenvironment can include physical features, such as a physical object orsurface. An example of a physical environment is a physical forest thatincludes physical plants and animals. A person can directly sense and/orinteract with a physical environment through various means, such ashearing, sight, taste, touch, and smell. In contrast, a person can usean electronic device to interact with and/or sense an extended reality(XR) environment that is wholly or partially simulated. The XRenvironment can include mixed reality (MR) content, augmented reality(AR) content, virtual reality (VR) content, and/or the like. With an XRsystem, some of a person's physical motions, or representations thereof,can be tracked and, in response, characteristics of virtual objectssimulated in the XR environment can be adjusted in a manner thatcomplies with at least one law of physics. For instance, the XR systemcan detect the movement of a user's head and adjust graphical contentand auditory content presented to the user similar to how such views andsounds would change in a physical environment. In another example, theXR system can detect movement of an electronic device that presents theXR environment (e.g., a mobile phone, tablet, laptop, or the like) andadjust graphical content and auditory content presented to the usersimilar to how such views and sounds would change in a physicalenvironment. In some situations, the XR system can adjustcharacteristic(s) of graphical content in response to other inputs, suchas a representation of a physical motion (e.g., a vocal command).

Many different types of electronic systems can enable a user to interactwith and/or sense an XR environment. A non-exclusive list of examplesinclude heads-up displays (HUDs), head mountable systems,projection-based systems, windows or vehicle windshields havingintegrated display capability, displays formed as lenses to be placed onusers' eyes (e.g., contact lenses), headphones/earphones, input systemswith or without haptic feedback (e.g., wearable or handheldcontrollers), speaker arrays, smartphones, tablets, and desktop/laptopcomputers. A head mountable system can have one or more speaker(s) andan opaque display. Other head mountable systems can be configured toaccept an opaque external display (e.g., a smartphone). The headmountable system can include one or more image sensors to captureimages/video of the physical environment and/or one or more microphonesto capture audio of the physical environment. A head mountable systemmay have a transparent or translucent display, rather than an opaquedisplay. The transparent or translucent display can have a mediumthrough which light is directed to a user's eyes. The display mayutilize various display technologies, such as uLEDs, OLEDs, LEDs, liquidcrystal on silicon, laser scanning light source, digital lightprojection, or combinations thereof. An optical waveguide, an opticalreflector, a hologram medium, an optical combiner, combinations thereof,or other similar technologies can be used for the medium. In someimplementations, the transparent or translucent display can beselectively controlled to become opaque. Projection-based systems canutilize retinal projection technology that projects images onto users'retinas. Projection systems can also project virtual objects into thephysical environment (e.g., as a hologram or onto a physical surface).

In XR systems that have an opaque display or a transparent ortranslucent display that is controlled to become opaque (e.g., includingXR systems that implement a video pass-through electronic device), theuser may not have a direct view of some or all of their physical settingwhen the system/device is in use. In a VR system, the entirety of theuser's view is a virtual setting. In a pass-through AR system, the useris reliant on the cameras of the system to provide a video view of thesetting. Aspects of the subject technology can help maintain a view ofthe physical setting in scenarios in which the system has low battery,or in the event of a failure of some portion of the system that providesinformation for processing the video feed (e.g., eye tracking sensorsand/or depth sensors) and/or that the video feed processes (e.g., avideo graphics processor) from the cameras to provide the video view.

Implementations of the subject technology described herein providevarious features for ensuring that the user of an XR electronic deviceand/or a video pass-through electronic device receives a view of thesetting in the case of low battery or system or sub-system failure. Asdescribed in further detail hereinafter, one or morestructured-shutdown-mode versions of the video view can be provided inthe event of a low power or failure state of the device.

FIG. 1A and FIG. 1B depict exemplary system 100 for use in variousextended reality technologies.

In some examples, as illustrated in FIG. 1A, system 100 includes device100 a. Device 100 a includes various components, such as processor(s)102, RF circuitry(ies) 104, memory(ies) 106, image sensor(s) 108,orientation sensor(s) 110, microphone(s) 112, location sensor(s) 116,speaker(s) 118, display(s) 120, and touch-sensitive surface(s) 122.These components optionally communicate over communication bus(es) 150of device 100 a.

In some examples, elements of system 100 are implemented in a basestation device (e.g., a computing device, such as a remote server,mobile device, or laptop) and other elements of system 100 areimplemented in a second device (e.g., a head-mounted device). In someexamples, device 100 a is implemented in a base station device or asecond device.

As illustrated in FIG. 1B, in some examples, system 100 includes two (ormore) devices in communication, such as through a wired connection or awireless connection. First device 100 b (e.g., a base station device)includes processor(s) 102, RF circuitry(ies) 104, and memory(ies) 106.These components optionally communicate over communication bus(es) 150of device 100 b. Second device 100 c (e.g., a head-mounted device)includes various components, such as processor(s) 102, RF circuitry(ies)104, memory(ies) 106, image sensor(s) 108, orientation sensor(s) 110,microphone(s) 112, location sensor(s) 116, speaker(s) 118, display(s)120, and touch-sensitive surface(s) 122. These components optionallycommunicate over communication bus(es) 150 of device 100 c.

System 100 includes processor(s) 102 and memory(ies) 106. Processor(s)102 include one or more general processors, one or more graphicsprocessors, and/or one or more digital signal processors. In someexamples, memory(ies) 106 are one or more non-transitorycomputer-readable storage mediums (e.g., flash memory, random accessmemory) that store computer-readable instructions configured to beexecuted by processor(s) 102 to perform the techniques described below.

System 100 includes RF circuitry(ies) 104. RF circuitry(ies) 104optionally include circuitry for communicating with electronic devices,networks, such as the Internet, intranets, and/or a wireless network,such as cellular networks and wireless local area networks (LANs). RFcircuitry(ies) 104 optionally includes circuitry for communicating usingnear-field communication and/or short-range communication, such asBluetooth®.

System 100 includes display(s) 120. Display(s) 120 may have an opaquedisplay. Display(s) 120 may have a transparent or semi-transparentdisplay that may incorporate a substrate through which lightrepresentative of images is directed to an individual's eyes. Display(s)120 may incorporate LEDs, OLEDs, a digital light projector, a laserscanning light source, liquid crystal on silicon, or any combination ofthese technologies. The substrate through which the light is transmittedmay be a light waveguide, optical combiner, optical reflector,holographic substrate, or any combination of these substrates. In oneexample, the transparent or semi-transparent display may transitionselectively between an opaque state and a transparent orsemi-transparent state. Other examples of display(s) 120 include headsup displays, automotive windshields with the ability to displaygraphics, windows with the ability to display graphics, lenses with theability to display graphics, tablets, smartphones, and desktop or laptopcomputers. Alternatively, system 100 may be designed to receive anexternal display (e.g., a smartphone). In some examples, system 100 is aprojection-based system that uses retinal projection to project imagesonto an individual's retina or projects virtual objects into a physicalsetting (e.g., onto a physical surface or as a holograph).

In some examples, system 100 includes touch-sensitive surface(s) 122 forreceiving user inputs, such as tap inputs and swipe inputs. In someexamples, display(s) 120 and touch-sensitive surface(s) 122 formtouch-sensitive display(s).

System 100 includes image sensor(s) 108. Image sensors(s) 108 optionallyinclude one or more visible light image sensor, such as charged coupleddevice (CCD) sensors, and/or complementary metal-oxide-semiconductor(CMOS) sensors operable to obtain images of physical elements from thephysical setting. Image sensor(s) also optionally include one or moreinfrared (IR) sensor(s), such as a passive IR sensor or an active IRsensor, for detecting infrared light from the physical setting. Forexample, an active IR sensor includes an IR emitter, such as an IR dotemitter, for emitting infrared light into the physical setting. Imagesensor(s) 108 also optionally include one or more event camera(s)configured to capture movement of physical elements in the physicalsetting. Image sensor(s) 108 also optionally include one or more depthsensor(s) configured to detect the distance of physical elements fromsystem 100. In some examples, system 100 uses CCD sensors, eventcameras, and depth sensors in combination to detect the physical settingaround system 100. In some examples, image sensor(s) 108 include a firstimage sensor and a second image sensor. The first image sensor and thesecond image sensor are optionally configured to capture images ofphysical elements in the physical setting from two distinctperspectives. In some examples, system 100 uses image sensor(s) 108 toreceive user inputs, such as hand gestures. In some examples, system 100uses image sensor(s) 108 to detect the position and orientation ofsystem 100 and/or display(s) 120 in the physical setting. For example,system 100 uses image sensor(s) 108 to track the position andorientation of display(s) 120 relative to one or more fixed elements inthe physical setting.

In some examples, system 100 includes microphones(s) 112. System 100uses microphone(s) 112 to detect sound from the user and/or the physicalsetting of the user. In some examples, microphone(s) 112 includes anarray of microphones (including a plurality of microphones) thatoptionally operate in tandem, such as to identify ambient noise or tolocate the source of sound in space of the physical setting.

System 100 includes orientation sensor(s) 110 for detecting orientationand/or movement of system 100 and/or display(s) 120. For example, system100 uses orientation sensor(s) 110 to track changes in the positionand/or orientation of system 100 and/or display(s) 120, such as withrespect to physical elements in the physical setting. Orientationsensor(s) 110 optionally include one or more gyroscopes and/or one ormore accelerometers.

FIG. 2 illustrates an example physical setting of an electronic deviceimplemented as a video pass-through electronic device. In the example ofFIG. 2 , a physical setting 200 of an electronic device such aselectronic device 100 a includes a physical object 202 and a physicalobject 210. As shown, an eye 201 of a user can have a line of sight 203that, in the absence of the electronic device 100 a, would land on thephysical object 202. In this example, the physical object 210 would alsobe visible to the user even though the user's gaze is directed towardthe physical object 202.

As shown, the electronic device 100 a (e.g., including display 205 ofthe electronic device such as an implementation of display 120 of FIGS.1A and 1B) blocks a direct view of physical objects 202 and 210 by theuser. In order to provide the user with a view of the physical objects202 and 210 (and/or other portions of the physical setting) that areblocked by the presence of the electronic device 100 a and the display205, a video view (e.g., a video pass-through view) of the physicalsetting 200 may be displayed by the display 205. For example, one ormore cameras of the electronic device 100 a may capture images of thephysical setting that can be displayed by the display 205.

In the example of FIG. 2 , a video view of at least the portion of thephysical setting 200 that is blocked from the user's direct view byelectronic device 100 a is displayed to the user by display 205,including a video version 204 of physical object 202, and a videoversion 208 of physical object 210. As shown, electronic device 100 amay also generate and display virtual content 212 that can be overlaidor otherwise added to and/or merged with the video view of the physicalsetting, to generate an extended reality setting for the user of theelectronic device.

In the example of FIG. 2 , the video version 204 of physical object 202and the video version 208 of physical object 210 are displayed bydisplay 205 to appear to the user to be at the actual physical locations(e.g., at the correct angular location and depth) of the physicalobjects 202 and 210, as they would be viewed by the user directly (e.g.,in the absence of electronic device 100 a). For example, electronicdevice 100 a may obtain a depth map of a portion of the physical setting200 using a depth sensor 129 (e.g., implementing one or more of imagesensor(s) 108, orientation sensor(s) 110, and/or location sensor(s) 116of FIGS. 1A and 1B), and may include one or more additional camerasand/or sensors such as eye sensor 222 that track the gaze location ofeach of the user's eyes and/or the location at which the gaze directionsof the user's eye converge (e.g., at a gaze plane). The depth map of thephysical setting 200, and the current eye data for the user's eye(s)201, can be used to perform a pixel level transformation (warping orreprojection) of the image frames from camera(s) 119 to generate displayframes from the correct viewpoint and position of the user's eyes at anygiven time, and with each pixel reprojected at the correct depthaccording to the depth map.

For example, to provide the most realistic experience for a user, theview of the physical setting 200 that is displayed by display 205 wouldinclude all of the rays of light from the physical setting that wouldenter the user's eye in the absence of electronic device 100 a (e.g., ifthe display 205 and/or other portions of the electronic device 100 awere not blocking the user's direct view of the physical setting).However, as illustrated in FIG. 2 , one or more of the cameras 119 thatcollect the light from the physical setting 200 may be offset from thelocation and/or gaze direction of the user's eye(s) 201. That is, theline of sight 203 from the user's eye 201 to a physical object 202 atany given time (e.g., based on the gaze direction of each eye at thattime) can be different from the line(s) of sight 206 of the camera(s).

Thus, the camera view of the physical setting can be warped (e.g.transformed, such as based on the current eye data from eye sensor 222and the depth map generated using sensors 129, using a reprojection ofeach pixel) by the electronic device 100 a to display the physicalsetting and the physical objects therein, to the user, from the user'sperspective.

The fields of view of any one of camera(s) 119 may be also limited ordifferent in comparison with the field of view of a human user. Forexample, portions of the physical setting in the user's peripheralvision may not be within the field of view of a color camera of theelectronic device. As another example, the warping of the image data totransform the image data to the user's perspective (e.g., to a currentline of sight 203 as determined by eye sensor 222) can cause portions ofan object (e.g., a portion along an edge of an object) that would bevisible to the user, to be displayed incorrectly as those portions arenot visible from the perspective of the camera(s). In order to provide amore complete field of view of the physical setting to the user and/orcorrect for perspective differences with display 205, images (e.g., thecamera feed or video feed) from multiple cameras can be combined to formeach display frame that is displayed to the user by display 205.

In one or more implementations, the cameras 119 of electronic device 100a may include, for example, one or more primary cameras and one or moresecondary cameras. The secondary camera(s) may consume less power thanthe primary camera(s). For example, the primary camera(s) may includeone or more color cameras, and the secondary camera(s) may include oneor more monochrome (e.g., black and white) cameras. The primarycamera(s) may have a resolution that is higher than the resolution ofthe secondary camera(s). In one or more implementations, color imagesobtained by the primary camera(s) may be used (e.g., warped, processed,and/or rendered) to generate the video pass-through view of the physicalsetting that is displayed to the user, while (e.g., monochrome) imagesfrom the secondary camera(s) can be used for other tasks such as objectdetection and tracking. In one or more implementations, portions of the(e.g., monochrome) images that include portions of the physical setting200 that are not included in the primary camera (e.g., color) images(e.g., due to a different physical location and/or field of view of thesecondary camera(s) relative to the primary camera(s)) can also be usedto fill the missing portions of the primary camera field(s) of view. Forexample, FIG. 2 illustrates portions 221 (e.g., portions at or near theedge(s) of the user's field of view) and a portion 223 (e.g., a portionalong an edge of a displayed version 208 of a physical object 210) of adisplay frame that have been filled using monochrome camera images.

In order to provide a visually seamless video view of the physicalsetting to the user, the portions of monochrome images that are used tofill the missing portions of color camera field(s) of view can becolorized. Colorizing the portions of the monochrome images that areused to fill the missing portions of the color camera field(s) of viewcan include determining (e.g., by providing color information from thecolor images to a machine learning model at the electronic device) acolor for each of the pixels of the portions of the monochrome imagesthat are used to fill the missing portions of the color camera field(s)of view, based on the color of pixels in the color images. The displayedportions 221 and 223 can be displayed using the determined color(s).

Obtaining the eye tracking data and the depth map, processing the imagedata from cameras 119 to warp, fill, and/or colorize the image data,and/or generating and adding virtual content to the processed image datato generate each display frame can be processor and/or power intensiveoperations (e.g., that combine to consume power at a first powerconsumption rate during normal operations of the display). Performingthese processor and/or power intensive operations relies on availabilityof power (e.g., from a power source such as one or more batteries at theelectronic device) and/or various processing resources (e.g., generalprocessing resources and/or dedicated processing resources such assensors 129 and eye sensor 222, and video processing components and/orprocesses, rendering components and/or processes, virtual contentgenerating components and/or processes) of the device. For example, anelectronic device such as electronic device 100 a may include firstprocessing circuitry (e.g., an image signal processor (ISP) forstreaming camera data from camera(s) 119 to display 205) and secondprocessing circuitry (e.g., an additional video processing chip orprocessing core) that receives the camera data and the sensor data andgenerates the warped, filled, and/or colorized video view for display toa user.

In some circumstances, the processing resources can fail (e.g., due tomechanical, electrical, or software issues) and/or power can becomelimited and/or unavailable (e.g., due to a system fault and/or a lowbattery) causing the system to deactivate or limit some processingresources. During use of general electronic devices that display contentthat is unrelated to the physical setting, a low power state (e.g., astate in which power is limited, unavailable, and/or insufficient forprocessing and display of camera data and/or virtual content) or afailure state (e.g., a state in which one or more of the generalprocessing resources and/or dedicated processing resources such as videoprocessing components and/or processes, rendering components and/orprocesses, virtual content generating components and/or processes usedfor generating the display frames for display becomes corrupted orunavailable) may result in loss of function of the display, which may beinconvenient or undesirable to the user.

In an electronic device such as an electronic device 100 a in which thecameras and display of the electronic device are used, by the user, toview and/or navigate some or all of their own physical setting (e.g.,via a video view of the physical setting as illustrated in FIG. 2 ), alow power state or a failure state that results in loss of function ofthe display can be particularly undesirable. This can be particularlytrue when the portion of the setting being viewed by the user on thedisplay of the electronic device is blocked by the display and/or theelectronic device itself (e.g., by the user holding the device in frontof their eyes or wearing a head mounted system that partially orcompletely blocks their view of some or all of their physical setting).

Electronic devices disclosed herein, such as electronic device 100 a,implement various structured display shutdown measures to help maintainat least a version of a video view of the physical setting for user(e.g., at a second power consumption rate lower than the first powerconsumption rate), even in cases of a low power state or failure statefor the electronic device.

For example, in the configuration shown in FIG. 2 , an electronic device(e.g., electronic device 100 a, also referred to herein as a device) isprovided with a display 205 that is configured to display virtualcontent 212 to a user that is located in a physical setting 200. Asshown, the device also includes one or more cameras 119 configured toobtain a video feed of at least a portion of the physical setting 200.The device may process the video feed to generate a video view of theportion of the physical setting (e.g., a portion of the physical settingincluding one or more physical objects such as physical objects 202 and210 at various positions and/or depths), and provide the video viewcorresponding to the processed video feed to the display for displaywith the virtual content.

As described above in connection with FIG. 2 , the device may processthe video feed to generate the video view (e.g., in a normal mode ofoperation of the device) by warping the video feed to conform the videofeed from a view of the camera(s) 119 to a view of the user, andcombining the virtual content 212 with the warped video feed fordisplay. In one or more implementations, the device may obtain currenteye data (e.g., a line of sight 203 corresponding to a gaze directionfor one or both eyes of the user, and/or a gaze plane corresponding to aconvergence location for the gaze directions of the users eyes) for theuser from at least one eye sensor such as eye sensor 222 (e.g., an eyetracking sensor that includes a light source and/or a camera for eacheye of the user), and warp the video feed to the view of the user basedon the current eye data. Warping the video feed based on the current eyedata may include performing a per-pixel warping or reprojection of thevideo feed based on the current eye data and a depth map of the physicalsetting.

As described above in connection with FIG. 2 , the device may alsoprocess the video feed to generate the video view (e.g., an a normalmode of operation of the device) by obtaining a monochrome video feedfrom the at least one monochrome camera (e.g., a monochrome or black andwhite camera, such as a single color visible light or infrared camera),performing a color estimation for a portion of the monochrome video feedbased on the video feed from the color camera, and filling a portion(e.g., corresponding to the portions 221 and 223 of the displayed videoframes) of the video feed using the monochrome video feed and the colorestimation.

In one or more implementations, the device may identify displaymodification state (e.g., a reduced functionality state of the device,such as a low power state or a failure state for the device). Responsiveto identifying the display modification, the device may modify theprocessing of the video feed to provide (e.g., at the second powerconsumption rate lower than the first power consumption rate) astructured-shutdown-mode version of the video view of the portion of thephysical setting to the display 205 while the device is in the displaymodification state. The structured-shutdown-mode version of the videoview may be generated using reduced power and/or processing resourcesrelative to generating the video view. FIGS. 3 and 4 illustrate examplesof structured-shutdown-mode versions of a video view that can begenerated while the device is in the display modification state (e.g.,the low power state or the failure state). A device may provide onepossible structured-shutdown-mode version of the video view (e.g., oneof the examples described herein) or can provide multiple possiblestructured-shutdown-mode versions of the video view (e.g., each beingavailable for use depending on a severity of the low power state or thefailure state).

FIG. 3 illustrates an example of a structured-shutdown-mode version ofthe video view of FIG. 2 . The structured-shutdown-mode version of thevideo view may be generated, for example, in a display modificationstate (e.g., low power state or a failure state) in which the eyesensor(s) 222, depth sensor(s), and/or video graphics processor areunavailable. In the example of FIG. 3 , electronic device 100 a hasmodified the processing of the video feed to generate thestructured-shutdown-mode version of the video view by discontinuing thewarping of the video feed based on the current eye data, obtainingnominal eye data for the user, and performing a transformation of thevideo feed to a predefined surface (e.g., an optical surface such assurface 300) based on the nominal eye data. The transformation of thevideo feed to the predefined surface may be a homography transformation.

The nominal eye data may include an interpupillary distance, and anassumption of a neutral gaze position for the user's eyes (e.g., a gazeposition of the user's eyes under the assumption that the user islooking straight forward at the horizon). The interpupillary distance(IPD) may be obtained from device memory, such as from a last known IPDmeasurement from eye sensor 222, or from a hardware component (e.g., alinear encoder) of a head mounted system that physically positionsdisplay modules of the device.

In one example, during normal operations, an IPD measurement may be(e.g., periodically or continuously) determined based on eye trackingdata from the eye sensor 222, and stored in memory (e.g., at the displayor in memory for an ISP) for later use in a low power or failure stateof the electronic device that trigger a display modification state. Inanother example, the IPD may be determined based on a mechanical and/orelectronic setting of the physical distance between display modules foreach of the user's eyes (e.g., by the user or by the device), thephysical distance obtainable from the linear encoder for use as aninterpupillary distance for the user. The distance of the surface 300 atwhich structured-shutdown-mode version of the video view is projected inFIG. 3 can be determined based the IPD and an assumed eye position atneutral gaze (e.g., when the measurement of the actual gaze direction ofthe user's eyes is discontinued to save power and/or processing in thelow power or failure mode, or is not available due to the failure mode).

In the example of FIG. 3 , surface 300 is depicted as a planar opticalsurface. In other implementations, other optical surfaces can be usedfor the homography transformation, such as a spherical optical surfaceonto which the video pixels can be projected.

In one or more implementations, the transformation of the video feedillustrated in FIG. 3 can be a pixel-independent homographytransformation. For example, each pixel of a video frame can beprojected onto the same predetermined surface 300 regardless of thedepth of the physical object represented in that pixel, in contrast withthe warping shown in FIG. 2 , for which each pixel of a video frame isreprojected for the user based on the current gaze location of theuser's eyes and the depth of the physical object in that pixel, toproject the content of that pixel at correct optical depth.

As illustrated in FIG. 3 , when the video versions 204 and 208 ofphysical objects 202 and 210 are projected onto a common surface 300,instead of appearing to the user to be at the correct correspondingdepth in the physical setting and to have a three-dimensional shape(e.g., as in FIG. 2 ), the video versions 204 and 208 may appear to beflat objects 302 and 310 at a common optical depth. The transformationand projection onto the predetermined surface 300 as shown in FIG. 3 ,is thus a lower quality projection than the warping described inconnection with FIG. 2 .

However, because the determination of the IPD is only performed once(e.g., in comparison with the current eye tracking used for thewarping), and because the transformation to the predetermined surface isa pixel-independent computation, projecting the video view onto thepredetermined surface 300 as shown in FIG. 3 can be performed usingreduced power and/or processing resources. Accordingly, modifying thevideo processing to the homography transformation of FIG. 3 can provideone implementation of a structured-shutdown-mode version of a video feedthat can be provided when the device is in the display modificationstate (e.g., the low power state or the failure state).

FIG. 3 also illustrates other modifications that can be performed (e.g.,together with and/or instead of the transformation and projection ontothe predetermined surface 300) to generate a structured-shutdown-modeversion of a video view when the device is in the display modificationstate. For example, as shown in FIG. 3 , generating and displaying thevirtual content 212 may be discontinued in a structured-shutdown-modeversion of a video view. In one or more structured-shutdown-modeversions of the video view, the transformation and projection onto thepredetermined surface 300 can be performed without combining the virtualcontent with the transformed video feed as shown in FIG. 3 . In one ormore structured-shutdown-mode versions of the video view, the device maymodify the processing of the video feed to generate thestructured-shutdown-mode version of the video view by continuing toperform the warping of the video feed based on the current eye data(e.g., as in FIG. 2 ) without combining the virtual content with thewarped video feed.

FIG. 3 also illustrates how the device may modify the processing of thevideo feed to generate the structured-shutdown-mode version of the videoview by discontinuing the filling of the portions 221 and/or 223 usingdata from other sensors and/or cameras such as a monochrome camera,and/or the discontinuing the color estimation for the filled pixels. Inthe example of FIG. 3 , the color estimation is discontinued andgreyscale filling is provided in portions 221 and 223. In one or moreimplementations, the filling itself may also be discontinued.

In one or more implementations, the display modification state may be alow power state that triggers a switch to a structured-shutdown-modeversion of a video view of a device and that may be identified based onthe power level of the device itself (e.g., a battery charge level)and/or can be determined based on additional information. For example,the device may determine a position of the user (e.g., a seated positionor a standing position) using one or more sensors of the device. Thedevice may identify the low power state for the device based at least inpart based on the position of the user as detected by the device. Forexample, a power threshold for identifying the low power state, and thustriggering generation of a power saving structured-shutdown-mode versionof a video view, may be higher when the user is standing than when theuser is sitting. In this way, the device can continue operating in thenormal mode of operation longer if the user is in a seated or otherwisestationary position.

FIG. 4 illustrates another modification that can be performed togenerate a structured-shutdown-mode version of a video view when thedevice is in the display modification state (e.g., low power state orthe failure state). In the example of FIG. 4 , the device hasdiscontinued all transformation of the video feed from camera(s) 119,and displays the video feed unmodified from the camera(s). As shown inFIG. 4 , displaying the video feed unmodified may cause the videoversions 204 and 208 of physical objects 202 and 210 to appear atdisplaced locations relative to their actual physical locations (e.g.,due to the different line(s) of sight 206 of the camera(s) 119 relativeto the line of sight 203 of the user's eye). Thestructured-shutdown-mode version of the video view of FIG. 4 may be adistorted view that includes missing portions 421 of the view (e.g.,corresponding to areas outside the fields of view of the camera(s)) andmay thus be provided as a last resort structured-shutdown-mode versionof the video view prior to total shutdown of the device and/or thedisplay.

In various operational scenarios, any or all of thestructured-shutdown-mode versions of the video view described above inconnection with FIGS. 2, 3, and 4 may be used to reduce thefunctionality and/or power consumption rate of the display in astructured manner while continuing to provide a view of the user'sphysical setting to the user in any of various low power modes and/orfailure modes of the device. In scenarios in which the user does nottake action to address the display modification state of the device, thedevice may detect an imminent failure of the structured-shutdown-modeversion of the video view (e.g., due to further failures of processingsystems and/or components, and/or due to a power level that has fallenbelow a minimum level for operating the camera(s) and/or display). Thedevice may provide an alert (e.g., an audio alert, a tactile alert, avisual alert, and/or any other output of the device to alert the user)to the user to move the display to unblock the direct view due to theimminent failure (e.g., to remove a head mounted system form the user'shead before the display and/or the device power fails completely).

FIG. 5 illustrates a flow diagram of an example process 500 forproviding a structured-shutdown-mode video view of a physical setting ofan electronic device in accordance with implementations of the subjecttechnology. For explanatory purposes, the process 500 is primarilydescribed herein with reference to the electronic device 100 a of FIGS.1A and 1B. However, the process 500 is not limited to the electronicdevice 100 a of FIGS. 1A and 1B, and one or more blocks (or operations)of the process 500 may be performed by one or more other components ofother suitable devices, including the electronic device 100 b and/or theelectronic device 100 c. Further for explanatory purposes, some of theblocks of the process 500 are described herein as occurring in serial,or linearly. However, multiple blocks of the process 500 may occur inparallel. In addition, the blocks of the process 500 need not beperformed in the order shown and/or one or more blocks of the process500 need not be performed and/or can be replaced by other operations.

As illustrated in FIG. 5 , at block 502, virtual content (e.g., virtualcontent 212) is displayed to a user that is located in a physicalsetting, with a display (e.g., display 205) of an electronic device suchas electronic device 100 a. The virtual content may be generated by theelectronic device for display as part of an extended reality settingthat includes a video view of some or all of a physical setting.

At block 504, a video feed of at least a portion of the physical settingis obtained with a camera of the electronic device. The camera may be acolor camera or a monochrome camera. The electronic device may includeone or more cameras such as camera(s) 119 described herein and/or one ormore sensors such as sensors 129 and/or eye sensor 222 described herein.

At block 506, the video feed may be processed to generate a video viewof the portion of the physical setting (e.g., at a first powerconsumption rate). In one or more implementations, processing the videofeed to generate the video view may include obtaining current eye datafor the user from the at least one eye sensor, warping the video feed toconform the video feed from a view of the camera to a view of the userbased on the current eye data, and combining the virtual content withthe warped video feed for display (e.g., as described above inconnection with FIG. 2 ).

In one or more implementations, the camera may be a color camera, andprocessing the video feed to generate the video view may includeobtaining a monochrome video feed from at least one monochrome camera ofthe electronic device, performing a color estimation for a portion ofthe monochrome video feed based on the video feed from the color camera,and filling a portion of the video feed using the monochrome video feedand the color estimation (e.g., as described above in connection withFIG. 2 ). Various device operations for obtaining current eye data forthe user from the at least one eye sensor, warping the video feed toconform the video feed from a view of the camera to a view of the userbased on the current eye data, combining the virtual content with thewarped video feed for display, obtaining a monochrome video feed from atleast one monochrome camera of the electronic device, performing a colorestimation for a portion of the monochrome video feed based on the videofeed from the color camera, and/or filling a portion of the video feedusing the monochrome video feed and the color estimation, can eachconsume device power that, in combination, results in power consumptionat the first power consumption rate.

At block 508, the video view corresponding to the processed video feedmay be provided to the display for display with the virtual content(e.g., as described above in connection with FIG. 2 ).

At block 510, a display modification state (e.g., a low power state or afailure state) may be identified for the device. Identifying the displaymodification state may include identifying a low power state which mayinclude determining that a charge level of a battery of the device isbelow a threshold (e.g., a fixed threshold or a threshold determinedbased on user position and/or movement). Identifying the displaymodification state may include identifying a failure state which mayinclude identifying a failure of a sensor, a camera, or a processingresource such as a video graphics processor.

At block 512, responsive to identifying the display modification state,the processing of the video feed may be modified to provide (e.g. at asecond power consumption rate lower than the first power consumptionrate) a structured-shutdown-mode version of the video view of theportion of the physical setting to the display while the device is inthe display modification state. In one or more implementations, when thestructured-shutdown-mode view is provided (e.g., in the displaymodification state), a notification may be displayed or otherwiseprovided to the user indicating that the device is in an adjustedoperation state, such as the display modification state. Modifying theprocessing of the video feed to provide the structured-shutdown-modeversion of the video view of the portion of the physical setting to thedisplay may include any or all of discontinuing a warping of the videofeed, discontinuing generating and displaying virtual content,discontinuing filling and/or color estimation operations, and/orproviding the video feed to the display unmodified, as described abovein connection with, for example, FIGS. 2, 3, and 4 . It is appreciatedthat discontinuing a warping of the video feed, discontinuing generatingand displaying virtual content, discontinuing filling and/or colorestimation operations, and/or providing the video feed to the displayunmodified can reduce the power consumption of the device, and result inproviding a version of the video feed for display at a second powerconsumption rate lower than the first power consumption rate.

In various implementations, modifying the processing of the video feedmay include providing the video feed as captured by the camera to thedisplay unmodified or discontinuing a warping operation, a fillingoperation, and/or a color estimation operation of the processing. In oneor more implementations, modifying the processing of the video feed toprovide the structured-shutdown-mode version of the video view of theportion of the physical setting to the display may include discontinuingthe warping of the video feed based on the current eye data; obtainingnominal eye data for the user; and performing a transformation of thevideo feed to a predefined surface based on the nominal eye data. In oneor more implementations, warping the video feed based on the current eyedata may include performing a per-pixel warping based on the current eyedata. Transforming of the video feed may include a pixel-independenttransformation, such as a transformation to a predetermined surface. Inone or more implementations, modifying the processing of the video feedto generate the structured-shutdown-mode version of the video viewincludes discontinuing at least one of the color estimation or thefilling.

Various processes defined herein consider the option of obtaining andutilizing a user's personal information. For example, such personalinformation may be utilized in order to provide a video view of aphysical setting. However, to the extent such personal information iscollected, such information should be obtained with the user's informedconsent. As described herein, the user should have knowledge of andcontrol over the use of their personal information.

Personal information will be utilized by appropriate parties only forlegitimate and reasonable purposes. Those parties utilizing suchinformation will adhere to privacy policies and practices that are atleast in accordance with appropriate laws and regulations. In addition,such policies are to be well-established, user-accessible, andrecognized as in compliance with or above governmental/industrystandards. Moreover, these parties will not distribute, sell, orotherwise share such information outside of any reasonable andlegitimate purposes.

Users may, however, limit the degree to which such parties may access orotherwise obtain personal information. For instance, settings or otherpreferences may be adjusted such that users can decide whether theirpersonal information can be accessed by various entities. Furthermore,while some features defined herein are described in the context of usingpersonal information, various aspects of these features can beimplemented without the need to use such information. As an example, ifuser preferences, account names, and/or location history are gathered,this information can be obscured or otherwise generalized such that theinformation does not identify the respective user.

These functions described above can be implemented in computer software,firmware or hardware. The techniques can be implemented using one ormore computer program products. Programmable processors and computerscan be included in or packaged as mobile devices. The processes andlogic flows can be performed by one or more programmable processors andby one or more programmable logic circuitry. General and special purposecomputing devices and storage devices can be interconnected throughcommunication networks.

Some implementations include electronic components, such asmicroprocessors, storage and memory that store computer programinstructions in a machine-readable or computer-readable medium (alsoreferred to as computer-readable storage media, machine-readable media,or machine-readable storage media). Some examples of suchcomputer-readable media include RAM, ROM, read-only compact discs(CD-ROM), recordable compact discs (CD-R), rewritable compact discs(CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layerDVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM,DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards,micro-SD cards, etc.), magnetic and/or solid state hard drives,read-only and recordable Blu-Ray® discs, ultra density optical discs,any other optical or magnetic media, and floppy disks. Thecomputer-readable media can store a computer program that is executableby at least one processing unit and includes sets of instructions forperforming various operations. Examples of computer programs or computercode include machine code, such as is produced by a compiler, and filesincluding higher-level code that are executed by a computer, anelectronic component, or a microprocessor using an interpreter.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some implementations areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some implementations, such integrated circuits executeinstructions that are stored on the circuit itself.

As used in this specification and any claims of this application, theterms “computer”, “server”, “processor”, and “memory” all refer toelectronic or other technological devices. These terms exclude people orgroups of people. For the purposes of the specification, the termsdisplay or displaying means displaying on an electronic device. As usedin this specification and any claims of this application, the terms“computer readable medium” and “computer readable media” are entirelyrestricted to tangible, physical objects that store information in aform that is readable by a computer. These terms exclude any wirelesssignals, wired download signals, and any other ephemeral signals.

To provide for interaction with a user, implementations of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; e.g., feedbackprovided to the user can be any form of sensory feedback, e.g., visualfeedback, auditory feedback, or tactile feedback; and input from theuser can be received in any form, including acoustic, speech, or tactileinput. In addition, a computer can interact with a user by sendingdocuments to and receiving documents from a device that is used by theuser; e.g., by sending web pages to a web browser on a user's clientdevice in response to requests received from the web browser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back end, middleware, or front end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), an inter-network (e.g., the Internet), andpeer-to-peer networks (e.g., ad hoc peer-to-peer networks).

The computing system can include clients and servers. A client andserver are generally remote from each other and may interact through acommunication network. The relationship of client and server arises byvirtue of computer programs running on the respective computers andhaving a client-server relationship to each other. In some embodiments,a server transmits data (e.g., an HTML page) to a client device (e.g.,for purposes of displaying data to and receiving user input from a userinteracting with the client device). Data generated at the client device(e.g., a result of the user interaction) can be received from the clientdevice at the server.

In accordance with aspects of the disclosure, a method is provided thatincludes obtaining a video feed of at least a portion of a physicalsetting, with a camera of an electronic device; processing the videofeed to generate a video view of the portion of the physical setting ata first power consumption rate; providing the video view correspondingto the processed video feed for display; identifying a displaymodification state for the device; and responsive to identifying thedisplay modification state, modifying the processing of the video feedto provide, at a second power consumption rate lower than the firstpower consumption rate, a structured-shutdown-mode version of the videoview of the portion of the physical setting for display while the deviceis in the display modification state.

In accordance with aspects of the disclosure, a device is provided thatincludes a display; a camera configured to obtain a video feed of atleast a portion of a physical setting; and one or more processorsconfigured to: process the video feed to generate a video view of theportion of the physical setting at a first power consumption rate;provide the video view corresponding to the processed video feed to thedisplay for display; identify a display modification state for thedevice; and responsive to identifying the display modification state,modify the processing of the video feed to provide, at a second powerconsumption rate lower than the first power consumption rate, astructured-shutdown-mode version of the video view of the portion of thephysical setting to the display while the device is in the displaymodification state.

In accordance with aspects of the disclosure, a non-transitorycomputer-readable medium is provided, the non-transitorycomputer-readable medium storing instructions that, when executed by oneor more processors of a device, cause the one or more processors to:obtain a video feed of at least a portion of a physical setting, with acamera of an electronic device; process the video feed to generate avideo view of the portion of the physical setting at a first powerconsumption rate; provide the video view corresponding to the processedvideo feed for display; identify a display modification state for thedevice; and responsive to identifying the display modification state,modify the processing of the video feed to provide, at a second powerconsumption rate lower than the first power consumption rate, astructured-shutdown-mode version of the video view of the portion of thephysical setting for display while the device is in the displaymodification state.

Those of skill in the art would appreciate that the various illustrativeblocks, modules, elements, components, methods, and algorithms describedherein may be implemented as electronic hardware, computer software, orcombinations of both. To illustrate this interchangeability of hardwareand software, various illustrative blocks, modules, elements,components, methods, and algorithms have been described above generallyin terms of their functionality. Whether such functionality isimplemented as hardware or software depends upon the particularapplication and design constraints imposed on the overall system. Thedescribed functionality may be implemented in varying ways for eachparticular application. Various components and blocks may be arrangeddifferently (e.g., arranged in a different order, or partitioned in adifferent way) all without departing from the scope of the subjecttechnology.

It is understood that the specific order or hierarchy of steps in theprocesses disclosed is an illustration of example approaches. Based upondesign preferences, it is understood that the specific order orhierarchy of steps in the processes may be rearranged. Some of the stepsmay be performed simultaneously. The accompanying method claims presentelements of the various steps in a sample order, and are not meant to belimited to the specific order or hierarchy presented.

The previous description is provided to enable any person skilled in theart to practice the various aspects described herein. The previousdescription provides various examples of the subject technology, and thesubject technology is not limited to these examples. Variousmodifications to these aspects will be readily apparent to those skilledin the art, and the generic principles defined herein may be applied toother aspects. Thus, the claims are not intended to be limited to theaspects shown herein, but is to be accorded the full scope consistentwith the language claims, wherein reference to an element in thesingular is not intended to mean “one and only one” unless specificallyso stated, but rather “one or more.” Unless specifically statedotherwise, the term “some” refers to one or more. Pronouns in themasculine (e.g., his) include the feminine and neuter gender (e.g., herand its) and vice versa. Headings and subheadings, if any, are used forconvenience only and do not limit the invention described herein.

The term website, as used herein, may include any aspect of a website,including one or more web pages, one or more servers used to host orstore web related content, etc. Accordingly, the term website may beused interchangeably with the terms web page and server. The predicatewords “configured to”, “operable to”, and “programmed to” do not implyany particular tangible or intangible modification of a subject, but,rather, are intended to be used interchangeably. For example, aprocessor configured to monitor and control an operation or a componentmay also mean the processor being programmed to monitor and control theoperation or the processor being operable to monitor and control theoperation. Likewise, a processor configured to execute code can beconstrued as a processor programmed to execute code or operable toexecute code.

The term automatic, as used herein, may include performance by acomputer or machine without user intervention; for example, byinstructions responsive to a predicate action by the computer or machineor other initiation mechanism. The word “example” is used herein to mean“serving as an example or illustration.” Any aspect or design describedherein as “example” is not necessarily to be construed as preferred oradvantageous over other aspects or designs.

A phrase such as an “aspect” does not imply that such aspect isessential to the subject technology or that such aspect applies to allconfigurations of the subject technology. A disclosure relating to anaspect may apply to all configurations, or one or more configurations.An aspect may provide one or more examples. A phrase such as an aspectmay refer to one or more aspects and vice versa. A phrase such as an“embodiment” does not imply that such embodiment is essential to thesubject technology or that such embodiment applies to all configurationsof the subject technology. A disclosure relating to an embodiment mayapply to all embodiments, or one or more embodiments. An embodiment mayprovide one or more examples. A phrase such as an “embodiment” may referto one or more embodiments and vice versa. A phrase such as a“configuration” does not imply that such configuration is essential tothe subject technology or that such configuration applies to allconfigurations of the subject technology. A disclosure relating to aconfiguration may apply to all configurations, or one or moreconfigurations. A configuration may provide one or more examples. Aphrase such as a “configuration” may refer to one or more configurationsand vice versa.

What is claimed is:
 1. A device, comprising: a display; and one or moreprocessors configured to: obtain a video feed of at least a portion of aphysical setting; process the video feed to generate a video view of theportion of the physical setting at a first power consumption rate;provide the video view corresponding to the processed video feed to thedisplay for display; identify a display modification state for thedevice; and responsive to identifying the display modification state,modify the processing of the video feed to provide, at a second powerconsumption rate lower than the first power consumption rate, astructured-shutdown-mode version of the video view of the portion of thephysical setting to the display while the device is in the displaymodification state.
 2. The device of claim 1, wherein the one or moreprocessors are configured to process the video feed to generate thevideo view by warping the video feed to conform the video feed from aview of a camera to a view of a user, wherein the display modificationstate comprises a low power state or a failure state, and wherein thestructured-shutdown-mode version of the video view comprises at least anun-warped version of the video view.
 3. The device of claim 2, furthercomprising at least one eye sensor, wherein the one or more processorsare further configured to: obtain current eye data for the user from theat least one eye sensor; and warp the video feed to the view of the userbased on the current eye data.
 4. The device of claim 3, wherein the oneor more processors are configured to modify the processing of the videofeed to generate the structured-shutdown-mode version of the video viewby: discontinuing the warping of the video feed based on the current eyedata; obtaining nominal eye data for the user; and performing atransformation of the video feed to a predefined surface based on thenominal eye data.
 5. The device of claim 4, wherein the current eye datacomprises a gaze direction for each eye of the user, and wherein thenominal eye data comprises an interpupillary distance and a neutralgaze.
 6. The device of claim 4, wherein the one or more processors areconfigured to warp the video feed based on the current eye data byperforming a per-pixel warping based on the current eye data and a depthmap, and wherein the transformation of the video feed comprises apixel-independent transformation.
 7. The device of claim 3, wherein theone or more processors are configured to process the video feed togenerate the video view by combining virtual content with the warpedvideo feed for display, and to modify the processing of the video feedto generate the structured-shutdown-mode version of the video view bycontinuing to perform the warping of the video feed based on the currenteye data without combining the virtual content with the warped videofeed.
 8. The device of claim 1, further comprising a color cameraconfigured to generate the video feed and at least one monochromecamera, and wherein the one or more processors are configured to processthe video feed to generate the video view by: obtaining a monochromevideo feed from the at least one monochrome camera; performing a colorestimation for a portion of the monochrome video feed based on the videofeed from the color camera; and filling a portion of the video feedusing the monochrome video feed and the color estimation.
 9. The deviceof claim 8, wherein the one or more processors are configured to modifythe processing of the video feed to generate thestructured-shutdown-mode version of the video view by discontinuing atleast one of the color estimation or the filling.
 10. The device ofclaim 1, wherein the display is configured to block a direct view of atleast a portion of the physical setting from a user when the device isworn by the user, and wherein the one or more processors are furtherconfigured to: detect an imminent failure of thestructured-shutdown-mode version of the video view; and provide an alertto the user to move the display to unblock the direct view due to theimminent failure.
 11. The device of claim 1, wherein the one or moreprocessors is configured to identify the display modification state atleast in part by identifying a low power state for the device based atleast in part based on a position of a user as detected by the device.12. A method, comprising: obtaining a video feed of at least a portionof a physical setting, with an electronic device; processing the videofeed to generate a video view of the portion of the physical setting ata first power consumption rate; providing the video view correspondingto the processed video feed for display; identifying a displaymodification state for the electronic device; and responsive toidentifying the display modification state, modifying the processing ofthe video feed to provide, at a second power consumption rate lower thanthe first power consumption rate, a structured-shutdown-mode version ofthe video view of the portion of the physical setting for display whilethe electronic device is in the display modification state.
 13. Themethod of claim 12, wherein processing the video feed to generate thevideo view includes: obtaining current eye data for a user from at leastone eye sensor; and warping the video feed to conform the video feedfrom a view of a camera to a view of the user based on the current eyedata.
 14. The method of claim 13, wherein modifying the processing ofthe video feed to generate the structured-shutdown-mode version of thevideo view includes: discontinuing the warping of the video feed basedon the current eye data; obtaining nominal eye data for the user; andperforming a transformation of the video feed to a predefined surfacebased on the nominal eye data.
 15. The method of claim 14, whereinwarping the video feed based on the current eye data comprisesperforming a per-pixel warping based on the current eye data and a depthmap, and wherein the transformation of the video feed comprises apixel-independent transformation.
 16. The method of claim 12, whereinobtaining the video feed comprises obtaining the video feed from a colorcamera of the electronic device, and wherein processing the video feedto generate the video view includes: obtaining a monochrome video feedfrom at least one monochrome camera of the electronic device; performinga color estimation for a portion of the monochrome video feed based onthe video feed from the color camera; and filling a portion of the videofeed using the monochrome video feed and the color estimation.
 17. Themethod of claim 16, wherein modifying the processing of the video feedto generate the structured-shutdown-mode version of the video viewincludes discontinuing at least one of the color estimation or thefilling.
 18. A non-transitory computer-readable medium storinginstructions that, when executed by one or more processors of a device,cause the one or more processors to: obtain a video feed of at least aportion of a physical setting, with an electronic device; process thevideo feed to generate a video view of the portion of the physicalsetting at a first power consumption rate; provide the video viewcorresponding to the processed video feed for display; identify adisplay modification state for the electronic device; and responsive toidentifying the display modification state, modify the processing of thevideo feed to provide, at a second power consumption rate lower than thefirst power consumption rate, a structured-shutdown-mode version of thevideo view of the portion of the physical setting for display while theelectronic device is in the display modification state.
 19. Thenon-transitory computer-readable medium of claim 18, wherein theinstructions, when executed by the one or more processors of the device,cause the one or more processors to modify the processing of the videofeed by providing the video feed as captured by a camera of theelectronic device to the display unmodified.
 20. The non-transitorycomputer-readable medium of claim 18, wherein the instructions, whenexecuted by the one or more processors of a device, cause the one ormore processors to modify the processing of the video feed bydiscontinuing a warping operation, a filling operation, or a colorestimation operation of the processing.